A First Attempt to Computing Generic Set Partitions: Delegation to an SQL Query Engine

Partitions are a very common and useful way of organizing data, in data engineering and data mining. However, partitions currently lack efficient and generic data management functionalities. This paper proposes advances in the understanding of this problem, as well as elements for solving it. We formulate the task as efficient processing, evaluating and optimizing queries over set partitions, in the setting of relational databases. However, producing universally fast execution plans remains a challenging task, since the underlying relational model has a significant impact on the algebraic definition of the operators and therefore on their implementation in terms of space and time costs. We first demonstrate that there is no trivial relational modeling for managing collections of partitions. We formally motivate a relational encoding and show that one cannot express all the operators of the partition lattice and set-theoretic operations as queries of the relational algebra. We investigate SQL features beyond FO to build optimized queries for partition operators. We provide multiple evidence of the inefficiency of FO queries. Our experimental results enforce this evidence, event accounting for careful SQL query optimization. We claim that there is a strong requirement for the design of a dedicated system to manage set partitions, or at least to supplement an existing data management system, to which both data persistence and query processing could be delegated.